Background: Endogenous estrogens may play a vital role in colorectal tumorigenesis. Estrogen receptor beta is the predominant subtype which mediates the biological effect of estrogens, while loss of expression of estrogen receptor beta has been indicated as a common step in the development of colorectal cancer (CRC). Epidemiological studies have revealed several functional polymorphisms of estrogen receptor beta (ESR2) for cancer risk, but relevant study in CRC is limited, particularly in men. This study aimed to investigate the association of circulating estradiol and variations of ESR2 with CRC risk in men. Methods: We initiated a case–control study consisting of 390 patients with CRC and 445 healthy controls in men only. We genotyped ESR2 single nucleotide polymorphisms (SNPs) rs1256049 and rs4986938 and measured serum estradiol concentration using chemilluminescence immunoassay. Multivariable logistic regression model was performed to evaluate the associations between these variables and CRC risk. Results: ESR2 rs1256049 CT/TT genotypes were associated with reduced risk of CRC (odds ratio [OR], 0.7, 95% confidence interval [CI], 0.5–1.0), while rs4986938 CT/TT genotypes were associated with increased risk of CRC (OR, 1.5, 95% CI, 1.0–2.1). In addition, the CRC risk increased with the number of risk genotypes of these two SNPs in a dose–response manner (Ptrend, 0.003). Specifically, subjects carrying risk genotypes of both SNPs had the highest risk of CRC (OR, 2.0, 95% CI, 1.3–3.3.). Moreover, serum estradiol concentration alone was associated with risk of CRC in men (OR, 1.2, 95% CI, 1.0–1.3). However, individuals presenting both rs4986938 CT/TT genotypes and high level of serum estradiol had a high risk of CRC (OR, 2.3, 95% CI, 1.4–3.9), compared with those presenting CC genotype and low level of serum estradiol. The similar joint results were not observed for SNP rs1256049. Conclusions: These results suggest that endogenous estrogen and genetic variations in ESR2 may individually, or more likely jointly, affect CRC risk in male Han Chinese population, while larger studies are needed to validate our findings.